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Abstract 

We investigate the separability of several well known classes of sub- 
groups of the mapping class group of a surface. 

1 Introduction and main results 
1.1 Introduction 

A subgroup H < G is said to be separable in G if it can be expressed as 
an intersection of finite index subgroups of G. If the trivial subgroup {1} is 
separable, G is residually finite. More generally, if every finitely generated 
subgroup of G is separable, G is subgroup separable (or LERF). 

Subgroup separability has been an important tool in geometry; for example 
it often permits the lifting of a 7Ti-injective immersion to an embedding in a 
finite cover [3U| (see in addition ©, OHOl, [HUE and Algebraically, it 

can also be viewed as an indication of an abundance of finite index subgroups 
and a rich interaction of these subgroups with the finitely generated ones. This 
powerful property is generally difficult to establish and the class of groups known 
to be subgroup separable is small. It is a theorem of M. Hall ^T] that free groups 
are subgroup separable. Scott reproved this and the subgroup separability of 
surface groups [301 ■ More recently, Agol, Long, and Reid |2] proved the Bianchi 
groups are subgroup separable (see pQ, ^Sjj or ^E])- In contrast, the mapping 
class group Mod(5) of a finite type surface S is known not to be subgroup 
separable except in a few very special cases (see Section 0). Nevertheless, it is 
well-stocked with subgroups of finite index and many interesting subgroups of 
Mod(S) are separable. 

Theorem 1.1 Solvable subgroups ofMod(S') are separable. 

The proof we give is a variation on the main idea of |27| . One of the ingre- 
dients readily implies the following. 
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Theorem 1.2 The Torelli group and each term in the Johnson filtration of 
Mod(S') is separable. 

The next theorem gives a potentially large class of subgroups that are sep- 
arable. For the statement, set T$ = ni(S) 1 X n (Ts), the SL(n; C)-character 
variety, and for a subvariety V, Stab(V) and Triv(V) the set and pointwise 
stabilizers of V under the action of Mod(5). 

Theorem 1.3 For any number field k and proper k-algebraic subvariety V of 
X n (Ts), the subgroups St&b(V) and Triv(V) are separable in Mod(5). 

This result is analogous to the linear setting, where for a finitely generated 
subgroup A of a fc-algebraic linear group G, the subgroup A n H, for any 
algebraic subgroup H defined over any number field, is separable in A (see |2()| . 
E3, and 0). 

One application of Theorem 11.31 provides separability of the handlebody 
groups and the Heegaard groups. 

Corollary 1.4 The two handlebody groups Mod(S l , H), Modo(iS r , H) and any 
Heegaard group Mod(S l , M 3 ) are separable. 

Finally, a modest generalization of the proof of the residually finiteness of 
Mod (S) PO] bears our final result. 

Theorem 1.5 The stabilizers of multi-curves, or more generally geometric sub- 
groups of Mod(5) are separable. 
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2 Definitions and background 

In this article, we denote a closed orientable surface with genus g and n marked 
points by S, the surface minus the marked points by S 1 , and T$ = ni(S). We 
define the mapping class group to be the quotient Mod(5) = DifT 1 " (S)/DiSq (5), 
where Diff + (S') is the group of orientation preserving diffeomorphisms of S leav- 
ing marked points invariant and Diffo(S') is the component containing the iden- 
tity. It is often useful to note that the natural map from Mod(5) to Out(rg) is 
injective. Indeed, when n = and g > 0, the Dehn-Nielsen Theorem says that 
this is onto a subgroup of index 2. 

On occasion we will want to consider surfaces-with-boundary and denote 
these by S to distinguish them from closed surfaces. In this case we modify the 
definition of Diff^S) and demand that boundary components be fixed point- 
wise. 
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2.1 The Johnson filtration 



By a descending series on Ts, we mean a nested series {Cj}j>o of subnormal 
subgroups of Ts- When each term Cj is characteristic in Ts, we call Cj a 
descending characteristic series. From a descending characteristic series Cj on 
Tg, we obtain a descending series iV,- on Mod(5). Specifically, the canonical 
set of epimorphisms TTj : Ts — ► Ts/Cj induce a family of homomorphisms 
Pj : Mod(S) — > Out(rg/(7j), and the series Nj is given by kerpj. Following 
Farb, we refer to the descending series on Mod(5) induced by the lower central 
series on Ts as the Johnson filtration: 

Mod(S) > Ni > N 2 > N s > • • • > Nj > . . . . 

The first non-trivial term Ni — T is usually referred to as the Torelli subgroup 
and the second N% = K, the Johnson Kernel. 

Theorem 2.1 (Bass-Lubotzky; [5 ) The Johnson filtration {JVj}j>o on the 
group Mod(S) satisfies: 

(a) 

f|JV iC Z(Mod(5)). 
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(b) Nj/Nj + i is torsion free for j > 0. 

2.2 Solvable subgroups 

Using ideas from the Nielsen-Thurston classification for elements of Mod(5), 
Birman, Lubotzky, and McCarthy [7] show that virtually solvable subgroups 
are virtually abelian (with bounded rank). Ivanov |12| strengthened this to the 
following form. 

Theorem 2.2 (Birman-Lubotzky-McCarthy, Ivanov) There is a finite in- 
dex subgroup Mod (S) < Mod(5) so that for any virtually solvable subgroup 
G < Mod(iS), G [~l Mod (S) is free abelian with rank at most 3g — 3. 

2.3 Multi-curve stabilizers and geometric subgroups 

By a multi-curve, we mean the isotopy class of a closed embedded 1-manifold 
in S for which each component is non-peripheral and homotopically essential. 
Mod(iS) acts on the set of multi-curves, and we denote the stabilizer of a multi- 
curve A by Stab(A). 

Given a proper subsurface S C S, if each component of E is 7Ti-injective, 
we say that E is incompressible (which is equivalent to saying that each bound- 
ary component is homotopically non-trivial an non-peripheral). We will often 
consider subsurfaces as well defined up to isotopy without comment. 

An element <f> £ Mod(S') is called pure if for every finite set of </>-invariant 
connected incompressible subsurfaces {Ei, E„} of S, each Ej is invariant. A 
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subgroup A < Mod(S) is called pure if every element of A is pure. Ivanov [T2] 
has shown that there are finite index pure subgroups Mod (S) < Mod(S), and 
Theorem 12.21 holds for any such subgroup. We note that pure subgroups are, in 
particular, torsion free. 

If S C S is an incompressible subsurface, Paris and Rolfsen [21] have proven 
that the inclusion induces a homomorphism Mod(S) — ► Mod(5) which is in- 
jective, unless two components of the boundary of £ are isotopic in S. For a 
general incompressible surface £, the kernel of this homomorphism is contained 
in the center of Mod(S). Following |23], when E is incompressible, even if the 
groups only inject modulo centers, we call the image of Mod(E) in Mod(5) a 
geometric subgroup and denote it by G(E). 

2.4 Handlebody and Heegaard groups 

In Masur initiated the study of two subgroups of Mod(5) called the han- 
dlebody groups defined as follows. Let H be a handlebody and choose a dif- 
feomorphism S — ► dH. The first handlebody group Mod(S, H) is the group 
consisting of those automorphisms of S which extend over H. The second is 
the subgroup Modo (5, H ) of Mod(5, H) consisting of those elements which in- 
duce the identity outer automorphism on %x{H). A similar subgroup studied 
by Goeritz |SJ, Powell |25| . and recently Scharlemann |29) is the Heegaard group 
Mod(5, S 3 ). This is defined by choosing a diffeomorphism from S to a Heegaard 
surface in S 3 , then taking the group of automorphisms of S that extend to S 3 . 
There is nothing special about S 3 in this construction, so for any 3-manifold 
M and diffeomorphism from S to a Heegaard surface in M, we can define the 
Heegaard group of (S,M), denoted by Mod(S r , M), in an analogous fashion. 

We remark that in the present notation, we have suppressed the identification 
of S with the boundary of the handlebody or Heegaard surface. 

2.5 Representation and character varieties 

Given a finitely generated group A and natural number n, the set Hom(A; SL(n; C)) 
is called the SL(n; C) -representation variety. This can be equipped with a nat- 
ural analytic structure (see PS]); in fact, the Z-algebraic structure on SL(n; C) 
provides Hom(A; SL(n; C)) with a Z-algebraic structure. SL(n; C) acts on 
Hom(A; SL(n; C)) by conjugation and the quotient (in the sense of geomet- 
ric invariant theory) is a set X n {A) called the SL(n; C)-character variety which 
is a Z-defined affine variety (see |23j). 

The inner automorphism group action of A on Hom(A; SL(n; C)) is absorbed 
by the action of SL(n;C). Hence the outer automorphism group Out(A) acts 
naturally on ^(A) by Z-algebraic automorphisms — see |19j . For any subvariety 
V C Xn(A) we define 



If k is a number field, Ok its ring of integers, and V is a /c-defined, then for any 
finite extension ring R/Ok, we let V(R) denote the set of i?-points of V. For 



Stab(V0 
Triv(V) 



{7 € Out(A) : 7 (V) C V} 
{7 e Out(A) : j\ v = idy}. 
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an ideal p < R, reducing the coordinates modulo p, we obtain a set we denote 
V(R/p). The reduction map (viewed as sets) r p : V(R) — > V(R/p) induces a 
homomorphism 

(r p )* : Aut(V(R)) — » Sym(V(R/p)), 

where Sym(V(iJ/p)) is the symmetric group on V(R/p). Note that since R/p 
is finite, Sym(V(R/p)) is a finite group. 

3 Subgroup separability 

3.1 Separability results 

For convenience we collect the requisite material on separability needed in this 
article here, referring the reader to the listed references for proofs. 

Lemma 3.1 (Long-Reid; [T7]) Let H < K < G be groups with [K : H] < oo 
and H separable in G. Then K is separable in G. 

Lemma 3.2 (Scott; |3D]) Let H,G < G, with [G : G ] < oo. Then H is 
separable in G if and only if H D Gq is separable in Gq . 

Lemma 3.3 Let G and H be a pair of groups and p : G — ► H a surjective 
homomorphism. If K < H is separable, then p _1 {K) is separable in G. 

Proof. The lemma is a consequence of the fact that finite index subgroups of H 
pull back to finite index subgroups of G under p. □ 

Theorem 3.4 (Grossman; \W\ ) Mod(5) is residually finite. 

One proof of this theorem uses the fact that Tg satisfies a strong type of 
residual finiteness: A group G is said to be conjugacy separable if for any two 
non-conjugate elements x,y € G, there is a homomorphism from G to a finite 
group for which the images of x and y remain non-conjugate. 

Theorem 3.5 (Stebe; [32 ) T$ is conjugacy separable. 

3.2 Lattices and nilpotent groups 

One key result of Segal |3J that we utilize is the following. 

Theorem 3.6 (Segal) The outer automorphism group of a finitely generated 
nilpotent group is a finite extension of an arithmetic lattice. 

The work in j22) (see also 0) provides the necessary counterpart to this 
theorem. 

Theorem 3.7 Polycyclic subgroups of arithmetic lattices are separable. 

In combination, these results produce the needed result. 

Theorem 3.8 Polycyclic subgroups of Out{Y s I C j) are separable, where Cj is 
the jth term of the lower central series ofTg. 

Using the residual finiteness of Out(F s/Cj), it is elementary to deduce The- 
orem ^21 
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4 Solvable subgroup separability in Mod(S') 



In this section, we prove Theorem 11.11 Our approach is a variation of the 
one taken in |27] for Aut(rg). By Theorem 12.21 it suffices to separate abelian 
subgroups. To this end, we prove the following proposition. 

Proposition 4.1 For every torsion free abelian subgroup A ofMod(S), there 
exists j (A) — j such that the quotient homomorphism pj : Mod(S) — ► Mod(S)/Nj 
restricted to A is an injection. 

Proof. To begin, set At = ker p l C\ A = N l C\ A. The inclusions N i+ i < N { 
produce inclusions A i+1 < Ai and so rankzA > rankzA i+ i. As {rankz-Aj} is a 
non- increasing sequence of non-negative integers, there exists j — j(A) > such 
that for all i,k > j, rankz-A, = rankzA.^. By Theorem 12.11 fa). PljAj = {1}, 
since Z(Mod(S)) is finite and A is torsion free. Consequently, it suffices to show 
that the sequence of groups {Ai} is eventually constant. Assuming otherwise, 
let i > j he such that Ai ^ Ai + \. By our selection of i, rankzA; = rankz^+i, 
and so Ai/A i+ i < Ni/N i+ i is a non-trivial finite group. However, this is in 
opposition with Theorem 12. II (b). □ 

Proof of Theorem Fix a pure subgroup Mod'(S') < Mod(S') with finite 

index. If A is any solvable subgroup of Mod(S'), then according to Theorem 
12.21 A = Aq (~l Mod' (S) is a torsion free abelian group. By Lemma \'S. 21 it suffices 
to separate A in Mod' (5). 

If we let N< = Ni n Mod'(S), then C\ t N[ = {1}; Z(Mod(5)) is finite and 
so intersects Mod'(S') trivially. In an abuse of notation, we write pi for the 
homomorphism from Mod (S) to Mod (S)/Nl, and note that Proposition 14.11 
still holds for some j = j(A). 

For any i > j, p^ 1 (p i (A)) = AN[ = A x N- since pi restricted to A is 
injective. An application of Theorem 13.81 implies Pi(A) is separable. Hence, 
by Lemma EP1 p~ 1 (p i (A)) is separable. The inclusion of A tx iV£ ^ A K iV t ' 
for k > i respects the semidirect product structure. This in combination with 
DiNi = {1} yields 

f]p^(p k (A)) = f)AKN'=A»<f)N' k = A. 

k>i k>i k>i 

Since p^ 1 (pk(A)) is separable, it is expressible as an intersection of finite index 
subgroups, hence so is A. □ 

This argument along with the theorem for Out(i 7 ' rl ) analogous to Theorem 
O-see 0— yields 

Theorem 4.2 Solvable subgroups of Out(F n ) are separable. 

The proof of Theorem 14.21 also requires an analog of Proposition 14.11 for 
polycyclic subgroups of Out(F„) which is achieved by considering Hirsch length 
as opposed to rank. That this yields all solvable subgroups of Out(F„) follows 
from the fact that solvable subgroups are virtually polycyclic — see 
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5 Subvariety stabilizer separability 



To prove Theorem 11.31 we would like to use the reduction maps from Section 
12.51 to construct finite index subgroups of Mod(S'). This, in turn, requires the 
existence of sufficiently many algebraic points on the variety V. The starting 
point for our proof of Theorem ll.3l is thus the following consequence of Hubert's 
Nullstellensatz. 

Proposition 5.1 Let V be a k-algebraic variety for a number field k. Then 

V(k) = |J V(K) 

k<K 
\K : k\ < oo 

is Zariski dense in V, where k is the algebraic closure ofk. 

Given a finite extension K/k and q <E V(K), it follows that q e V(R), 
for some finite extension ring R/Ok- For example, if m is the product of the 
denominators occurring in the coordinates of q, one can take R = 0#-[l/m]. 
Coupled with Proposition l5.il this produces a plethora of points algebraically 
defined in the variety V. 

Proof of Theorem MJft Let V C X n (Ts) be a fc-algebraic subvariety, and let 
a C oy C C[T] be the associated ideals for X n (Ts) and V, respectively. For 
7 e Mod(S') \ Stab(V), tner e exists q e V such that ^(q ) £ V. Since V(k) is 
Zariski dense in V, we can assume go £ V {%■)■> f° r some finite extension K/k. 
Indeed, from the discussion above, qo € V(R) for some finite extension ring 
R/Ok- Next, select a generating set fi, . . . ,f r 6 R[T] for ay. It follows that 
fj(qo)J 3 (-f(qo)) G R for all j. By assumption, -f(q ) £ V, and so fj(-f(q )) ^ 
for some generator . Thus, we can select an ideal p of R such that fj (7(170)) 7^ 
mod p. Since the points of V(R/p) are precisely those of the form r p (q), with 
f t (q) = mod p for each 1 < i < r, it follows that ^(7(170)) ^ V(R/p). 
Since (rp)*('j)(rp(qo)) = ^(7(^0)), we have (r p )*(7) £ Stab(V/p). On the other 
hand, because r p (V(R)) — V(R/p), we see that (rp)*(Stab(V)) is contained in 
Stab(V(i?/p)). Therefore, (r p )- 1 (Stab(V A (i?/p))) separates 7 from Stab(V). 

Next let 7 £ Mod(5')\Triv(V). As above, we may select q £ V(R), for some 
finite extension R/Ok, such that 7(90) 7^ qo- The separation of Triv(V") from 
7 is obtained in a similar fashion to the above by selecting an ideal p of R such 
that 7(<7o) 7^ Qo m od p. Specifically, with such an ideal, (r p )^ 1 (Triv(V(i?/p))) 
separates 7 and Triv(V). □ 

As mentioned in the introduction, separability of the handlebody groups and 
Heegaard groups follows from Theorem 1 1.31 

Proof of Corollary \1.4\ A Heegaard group is the intersection of two conjugates 
of the handlebody group Mod (5, H), so it suffice to prove the corollary for 
Mod^, H) and Mod (S", H). 
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We note that the embedding i : S — ► H induces an inclusion 

i* :M^i(H)) — » X 2 (T S ). 

The image under i* , denoted by Vjj, is Z-defined. Specifically, its obtained by 
declaring certain elements in Ts — those that bound disks in H — to be trivial. 
Clearly Mod{S,H) < Stab(Ver)- Mod(S) \ Mod(S,H) then it takes a 

simple closed curve that bounds a disk in H to one which does not bound a disk, 
so by Dehn's Lemma [2B1 it is not homotopically trivial in H . Since there exist 
faithful representations of TTi(-ff) to SL(2;C), it follows that <f> 4- Stab(V/j), 
hence Stab(Vff) = Mod(5,F), and Mod(S,H) is separable by Theorem Ol 
Similarly, Modo (5, H ) = Triv(Vjj) and so is separable by Theorem II. 31 □ 

Remark 5.2 The idea of using algebraic actions of groups on varieties to de- 
duce residual properties is not new. Bass and Lubotzky '4f used this to produce 
another proof of the residual finiteness of Mod(S). 

6 Geometric subgroups 

The next theorem easily implies Theorem ll.51 

Theorem 6.1 If A is any finite set of conjugacy classes in Ts, then the group 
Stab(A) is separable. 

Proof. Write the elements of A as A = {a x , . . . , a n } . Let <j) G Mod(5)\Stab(A) 
and Oi be such that = x and x ^ <Xj for any j. By Theorem l3.5l there exists 
a finite group F and epimorphism ip : Ts — > F for which rp(x) is not conjugate 
to tp(ai) for any i. Without loss of generality, we may assume that ker(?/>) is 
characteristic (if not, replace F by the quotient of Ts by the characteristic core 
of ker(-0)). The homomorphism ip induces a homomorphism 

ip* : Out(Ls) — > Out(F). 

The group Out(F) acts on the set of conjugacy classes of F and we can consider 
the subgroup Stab(V>(A)). Visibly, Stab(A) C V~ 1 (Stab(V'(A))) and does not 
contain , 0*( < / > ) since ip(x) is not in any of the conjugacy classes of ^>(A). □ 

Proof of Theorem \1.5[ The proof for multi-curve stabilizers is immediate from 
Theorem I6.ll Suppose S C S is an incompressible subsurface. To separate 
G(E) we show that for any finite index pure subgroup Mod' (5) < Mod(S') 

G(E) n Mod'(S) = Stab(A) n Mod'(S*) 

for some multi-curve A. Then, by Lemma \'6.2\ and Theorem ltj.il G(E) will be 
separable. 

We construct A as follows. Let 9E denote the union of the conjugacy classes 
in Ts representing the boundary components of £ and Ri,...,Rk denote those 
components of the complementary subsurface S \ £ that are not homeomorphic 
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to annuli or pairs of pants. For each Ri, let 0Ci,f3i be a pair of simple closed 
curves which bind Ri (meaning every other non-peripheral essential closed curve 
on Ri intersects one of on or fy). Any automorphism of Ri that fixes both on 
and Pi must have finite order (up to Dehn twisting in curves parallel to the 
boundary) — compare ^3]. Finally, set A = <9E U {ax, Pi, ...,au, fik}- 

If <p is any pure automorphism which preserves A, it must be the identity 
on S \ E. This is true for each Ri by the discussion in the previous paragraph, 
the definition of pure automorphism, and the fact that Dehn twisting in the 
boundary of S \ E can be absorbed into E. On the other hand, since the auto- 
morphism group of an annulus or pair of pants is finite, modulo Dehn twisting in 
the boundary, the purity assumption shows that it holds for these components as 
well. Thus Stab(A) nMod'(S) < G(E) n Mod'(S'). Since being in Stab(A) puts 
absolutely no constraint on the restriction to E, the other inclusion evidently 
holds. □ 

Remark 6.2 It is worth mentioning that conjugacy separability is equivalent 
to the quotient topology on the set of conjugacy classes induced by the profinite 
topology on Ts being Hausdorff. Consequently, finite sets are closed. Using the 
proof of Theorem I6.il as a guide, one can prove: 

Theorem 6.3 The stabilizer in Mod(S') of any closed set of conjugacy classes 
in Ts is separable. 

7 Appendix: Mod(S') is not subgroup separable 

For completeness, in this section we prove that Mod(.S') is not subgroup sep- 
arable. The idea is to show that Mod(S) contains a subgroup which is not 
subgroup separable. To begin, whenever S contains two disjoint incompressible 
subsurfaces more complicated than a pair of pants, Mod(S') contains an isomor- 
phic copy of F 2 x F 2 , and so cannot be subgroup separable. In the remaining 
cases, Mod(S') is virtually free, and hence obviously separable, or else S is a 
torus with two marked points or a sphere with 5 marked points. In these cases, 
Mod(S') contains the fundamental group of every 3-manifold fibered over the 
circle with fiber a once punctured torus or 4-punctured sphere, respectively. 
It is well known that there are such 3-manifold groups (with reducible mon- 
odromy) which arc not subgroup separable (see [Sj). Therefore, the mapping 
class groups in these special cases are not subgroup separable. 
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